Redis, JSON vs MSGPACK

1. 개요

JVM 환경의 Webflux 서버에서 Redis 사용시 JSON 을 사용할때와 MessagePack 을 사용할때 어떤 점이 다른지 확인하고자 하며 아래와 같이 구현하고 벤치마크 하였다.

2. 구현

1) Dependency

implementation 'org.msgpack:jackson-dataformat-msgpack:0.9.4'

2) Configure

Java-based configuration 으로 ReactiveRedisTemplate 를 생성하는 메소드를 등록한다. Key 는 String 으로 직렬화하고 Value 는 사용 클래스에서 JSON 인지 MSGPACK 인지에 따라 ObjectMapper 클래스로 변환해서 저장하거나 로드하기로 했다.

@Configuration
public class RedisConfig {

    @Bean
    public ReactiveRedisTemplate<String, Object> getReactiveRedisTemplate(
            ReactiveRedisConnectionFactory reactiveRedisConnectionFactory
    ) {
        RedisSerializationContext<String, Object> serializationContext =
                RedisSerializationContext.<String, Object>newSerializationContext(new StringRedisSerializer())
                        .key(new StringRedisSerializer())
                        .hashKey(new StringRedisSerializer())
                        .build();

        return new ReactiveRedisTemplate<>(reactiveRedisConnectionFactory, serializationContext);
    }
}

3) JSON 을 위한 구현

@Service
public class BoardServiceDODB implements BoardDocumentService {

    private final BoardRepositoryDODB boardRepository;
    private final BoardDocumentMapper boardMapper;
    private final ReactiveHashOperations<String, String, String> redisHash;
    private final ReactiveZSetOperations<String, String> redisZSet;

    // JSON 사용시 ObjectMapper 에 특별히 JSONFactory를 지정하지 않는다
    private final ObjectMapper objectMapper = new ObjectMapper(
    );

    private final ReactiveKafkaProducerTemplate<String, BoardDocumentDTO.Create> kafkaSender;


    @Autowired
    public BoardServiceDODB(
            BoardRepositoryDODB boardRepository,
            BoardDocumentMapper boardMapper,
            ReactiveRedisTemplate<String, String> getReactiveRedisTemplate,
            ReactiveKafkaProducerTemplate<String, BoardDocumentDTO.Create> getKafkaProducerTemplate
    ) {
        this.boardRepository = boardRepository;
        this.boardMapper = boardMapper;

        this.redisHash = getReactiveRedisTemplate.opsForHash();
        this.redisZSet = getReactiveRedisTemplate.opsForZSet();

        this.kafkaSender = getKafkaProducerTemplate;

        // LocalDateTime을 올바르게 JSON에 저장하고 읽기 위해 필요한 부분
        this.objectMapper.registerModule(new JavaTimeModule());
    }

    @Override
    public Mono<BoardDocumentDTO.Create> CreateItem(BoardDocumentDTO.Create dto) {
        return redisHash.increment("counter", "board", 1)
                .flatMap((Long counter) -> {
                    LocalDateTime now = LocalDateTime.now();
                    dto.setScore(counter);
                    dto.setModifiedAt(now);
                    dto.setCreatedAt(now);
                    dto.setCreatedTs(System.currentTimeMillis());
                    return boardRepository.save(boardMapper.ItemToEntity(dto));
                })
                .flatMap(board -> {
                    dto.setId(board.getId());

                    String encode;
                    try {
                        // objectMapper 로 오브젝트 데이터를 String 으로 전환
                        encode = objectMapper.writeValueAsString(dto);
                    } catch (JsonProcessingException e) {
                        return Mono.error(new RuntimeException(e));
                    }
                    return redisZSet.add("board", encode, dto.getScore())
                            .flatMap(bool -> {
                                if (bool) {
                                    return Mono.just(dto);
                                } else {
                                    return Mono.error(new RuntimeException("redis zset add error"));
                                }
                            });
                })
                ;
    }

    @Override
    public Flux<BoardDocumentDTO.Create> SelectCacheList(Integer page, Integer size) {
        return redisZSet.reverseRangeByScore(
                        "board",
                        Range.unbounded(),
                        Limit.limit().offset((page - 1) * size).count(size)
                )
                .flatMap(object -> {
                    try {
                        // objectMapper 로 전달받은 String 을 Object 로 전환
                        return Mono.just(objectMapper.readValue(object, BoardDocumentDTO.Create.class));
                    } catch (Exception e) {
                        return Mono.error(new RuntimeException(e));
                    }
                })
                .switchIfEmpty(
                        boardRepository.findAll().flatMap(board -> {
                            return Mono.just(boardMapper.ToCreateDTO(board));
                        })
                );
    }

}

4) MSGPACK 을 위한 구현

@Service
public class BoardServiceDODB implements BoardDocumentService {

    private final BoardRepositoryDODB boardRepository;
    private final BoardDocumentMapper boardMapper;
    private final ReactiveHashOperations<String, String, Object> redisHash;
    private final ReactiveZSetOperations<String, Object> redisZSet;

    // JSON 사용시 ObjectMapper 에 MessagePackFactory 지정한다
    private final ObjectMapper objectMapper = new ObjectMapper(
            new MessagePackFactory()
    );

    private final ReactiveKafkaProducerTemplate<String, BoardDocumentDTO.Create> kafkaSender;


    @Autowired
    public BoardServiceDODB(
            BoardRepositoryDODB boardRepository,
            BoardDocumentMapper boardMapper,
            ReactiveRedisTemplate<String, Object> getReactiveRedisTemplate,
            ReactiveKafkaProducerTemplate<String, BoardDocumentDTO.Create> getKafkaProducerTemplate
    ) {
        this.boardRepository = boardRepository;
        this.boardMapper = boardMapper;

        this.redisHash = getReactiveRedisTemplate.opsForHash();
        this.redisZSet = getReactiveRedisTemplate.opsForZSet();

        this.kafkaSender = getKafkaProducerTemplate;

        this.objectMapper.registerModule(new JavaTimeModule());
    }

    @Override
    public Mono<BoardDocumentDTO.Create> CreateItem(BoardDocumentDTO.Create dto) {
        return redisHash.increment("counter", "board", 1)
                .flatMap((Long counter) -> {
                    LocalDateTime now = LocalDateTime.now();
                    dto.setScore(counter);
                    dto.setModifiedAt(now);
                    dto.setCreatedAt(now);
                    dto.setCreatedTs(System.currentTimeMillis());
                    return boardRepository.save(boardMapper.ItemToEntity(dto));
                })
                .flatMap(board -> {
                    dto.setId(board.getId());

                    Object encode;
                    try {
                        encode = objectMapper.writeValueAsBytes(board);
                    } catch (JsonProcessingException e) {
                        return Mono.error(new RuntimeException(e));
                    }
                    return redisZSet.add("board", encode, dto.getScore())
                            .flatMap(bool -> {
                                if (bool) {
                                    return Mono.just(dto);
                                } else {
                                    return Mono.error(new RuntimeException("redis zset add error"));
                                }
                            });
                })
                ;
    }

    @Override
    public Flux<BoardDocumentDTO.Create> SelectCacheList(Integer page, Integer size) {
        return redisZSet.reverseRangeByScore(
                        "board",
                        Range.unbounded(),
                        Limit.limit().offset((page - 1) * size).count(size)
                )
                .flatMap(object -> {
                    try {
                        return Mono.just(objectMapper.readValue((byte[]) object, BoardDocumentDTO.Create.class));
                    } catch (Exception e) {
                        return Mono.error(new RuntimeException(e));
                    }
                })
                .switchIfEmpty(
                        boardRepository.findAll().flatMap(board -> {
                            return Mono.just(boardMapper.ToCreateDTO(board));
                        })
                );
    }

}

3. 벤치마크 기준

1) 벤치마크 부하

locust -H http://127.0.0.1:9090 -u1000 -r 200 -t 60s --autostart -f tests/locust_board_select.py

2) 쓰기 데이터

아래와 같은 양식의 데이터를 랜덤으로 생성해서 무작위 삽입한다.

{
  "member_id": {
    "$numberLong": "60992"
  },
  "score": {
    "$numberLong": "3"
  },
  "title": "Elizabeth Norman",
  "body": "These religious theory thousand company girl long environment. Party family type. Phone according cup exist.\nService require network real somebody view. Sing woman five by.",
  "address": "7321 Michael Roads Apt. 058\nSmithfurt, CT 85072",
  "country": "Papua New Guinea",
  "job": "Product manager",
  "credit": "4716296968179136",
  "ssn": "333-96-3366",
  "company": "Diaz and Sons",
  "phone_number": "+1-610-354-4229x154",
  "modified_at": {
    "$date": "2023-11-28T01:15:04.126Z"
  },
  "created_at": {
    "$date": "2023-11-28T01:15:04.126Z"
  },
  "created_ts": {
    "$numberLong": "1701134104126"
  },
  "_class": "com.example.webfluxapi.entity.BoardDocument"
}

3) 읽기 데이터

아래와 같은 양식의 데이터를 다소 무거운 16.9KB 정도의 크기로 만들어 목록으로 반환한다.

{
    "code": 200,
    "message": "success",
    "data": [
        {
            "id": "65653f5000b66559c04fe767",
            "score": 120000,
            "member_id": 71642,
            "title": "Jamie Terrell",
            "body": "Five move if necessary final mind. Seem stop sound successful.\nYard positive clear beat building American whatever education.",
            "address": "98886 Johnson Squares Suite 099\nNorth Christophershire, VA 12430",
            "country": "Iran",
            "job": "Automotive engineer",
            "credit": "38926412620637",
            "ssn": "177-97-5494",
            "company": "Barnes Ltd",
            "phone_number": "941-836-2530x59041",
            "modified_at": "2023-11-28 10:16:00",
            "created_at": "2023-11-28 10:16:00",
            "created_ts": 1701134160136
        }, (생략)
    ]
}

4. JSON, 벤치마크 결​과

1) JSON, 저장 용량

2) JSON, 쓰기 성능

3) JSON, 읽기 성능

5. MSGPACK, 벤치마크 결​과

1) MSGPACK, 저장 용량

2) MSGPACK, 쓰기 성능

3) MSGPACK, 읽기 성능

6. 결과

JSON 으로 저장하는 것보다 MSGPACK 으로 저장하는 것이 더 효율이 좋고 MSGPACK 데이터를 읽어서 처리하는 기준에서도 조금 더 우세한 것으로 보인다.