Redis, JSON vs MSGPACK
1. 개요
JVM 환경의 Webflux 서버에서 Redis 사용시 JSON 을 사용할때와 MessagePack 을 사용할때 어떤 점이 다른지 확인하고자 하며 아래와 같이 구현하고 벤치마크 하였다.
2. 구현
1) Dependency
implementation 'org.msgpack:jackson-dataformat-msgpack:0.9.4'build.gradle
2) Configure
Java-based configuration 으로 ReactiveRedisTemplate 를 생성하는 메소드를 등록한다. Key 는 String 으로 직렬화하고 Value 는 사용 클래스에서 JSON 인지 MSGPACK 인지에 따라 ObjectMapper 클래스로 변환해서 저장하거나 로드하기로 했다.
@Configuration
public class RedisConfig {
@Bean
public ReactiveRedisTemplate<String, Object> getReactiveRedisTemplate(
ReactiveRedisConnectionFactory reactiveRedisConnectionFactory
) {
RedisSerializationContext<String, Object> serializationContext =
RedisSerializationContext.<String, Object>newSerializationContext(new StringRedisSerializer())
.key(new StringRedisSerializer())
.hashKey(new StringRedisSerializer())
.build();
return new ReactiveRedisTemplate<>(reactiveRedisConnectionFactory, serializationContext);
}
}
3) JSON 을 위한 구현
@Service
public class BoardServiceDODB implements BoardDocumentService {
private final BoardRepositoryDODB boardRepository;
private final BoardDocumentMapper boardMapper;
private final ReactiveHashOperations<String, String, String> redisHash;
private final ReactiveZSetOperations<String, String> redisZSet;
// JSON 사용시 ObjectMapper 에 특별히 JSONFactory를 지정하지 않는다
private final ObjectMapper objectMapper = new ObjectMapper(
);
private final ReactiveKafkaProducerTemplate<String, BoardDocumentDTO.Create> kafkaSender;
@Autowired
public BoardServiceDODB(
BoardRepositoryDODB boardRepository,
BoardDocumentMapper boardMapper,
ReactiveRedisTemplate<String, String> getReactiveRedisTemplate,
ReactiveKafkaProducerTemplate<String, BoardDocumentDTO.Create> getKafkaProducerTemplate
) {
this.boardRepository = boardRepository;
this.boardMapper = boardMapper;
this.redisHash = getReactiveRedisTemplate.opsForHash();
this.redisZSet = getReactiveRedisTemplate.opsForZSet();
this.kafkaSender = getKafkaProducerTemplate;
// LocalDateTime을 올바르게 JSON에 저장하고 읽기 위해 필요한 부분
this.objectMapper.registerModule(new JavaTimeModule());
}
@Override
public Mono<BoardDocumentDTO.Create> CreateItem(BoardDocumentDTO.Create dto) {
return redisHash.increment("counter", "board", 1)
.flatMap((Long counter) -> {
LocalDateTime now = LocalDateTime.now();
dto.setScore(counter);
dto.setModifiedAt(now);
dto.setCreatedAt(now);
dto.setCreatedTs(System.currentTimeMillis());
return boardRepository.save(boardMapper.ItemToEntity(dto));
})
.flatMap(board -> {
dto.setId(board.getId());
String encode;
try {
// objectMapper 로 오브젝트 데이터를 String 으로 전환
encode = objectMapper.writeValueAsString(dto);
} catch (JsonProcessingException e) {
return Mono.error(new RuntimeException(e));
}
return redisZSet.add("board", encode, dto.getScore())
.flatMap(bool -> {
if (bool) {
return Mono.just(dto);
} else {
return Mono.error(new RuntimeException("redis zset add error"));
}
});
})
;
}
@Override
public Flux<BoardDocumentDTO.Create> SelectCacheList(Integer page, Integer size) {
return redisZSet.reverseRangeByScore(
"board",
Range.unbounded(),
Limit.limit().offset((page - 1) * size).count(size)
)
.flatMap(object -> {
try {
// objectMapper 로 전달받은 String 을 Object 로 전환
return Mono.just(objectMapper.readValue(object, BoardDocumentDTO.Create.class));
} catch (Exception e) {
return Mono.error(new RuntimeException(e));
}
})
.switchIfEmpty(
boardRepository.findAll().flatMap(board -> {
return Mono.just(boardMapper.ToCreateDTO(board));
})
);
}
}
4) MSGPACK 을 위한 구현
@Service
public class BoardServiceDODB implements BoardDocumentService {
private final BoardRepositoryDODB boardRepository;
private final BoardDocumentMapper boardMapper;
private final ReactiveHashOperations<String, String, Object> redisHash;
private final ReactiveZSetOperations<String, Object> redisZSet;
// JSON 사용시 ObjectMapper 에 MessagePackFactory 지정한다
private final ObjectMapper objectMapper = new ObjectMapper(
new MessagePackFactory()
);
private final ReactiveKafkaProducerTemplate<String, BoardDocumentDTO.Create> kafkaSender;
@Autowired
public BoardServiceDODB(
BoardRepositoryDODB boardRepository,
BoardDocumentMapper boardMapper,
ReactiveRedisTemplate<String, Object> getReactiveRedisTemplate,
ReactiveKafkaProducerTemplate<String, BoardDocumentDTO.Create> getKafkaProducerTemplate
) {
this.boardRepository = boardRepository;
this.boardMapper = boardMapper;
this.redisHash = getReactiveRedisTemplate.opsForHash();
this.redisZSet = getReactiveRedisTemplate.opsForZSet();
this.kafkaSender = getKafkaProducerTemplate;
this.objectMapper.registerModule(new JavaTimeModule());
}
@Override
public Mono<BoardDocumentDTO.Create> CreateItem(BoardDocumentDTO.Create dto) {
return redisHash.increment("counter", "board", 1)
.flatMap((Long counter) -> {
LocalDateTime now = LocalDateTime.now();
dto.setScore(counter);
dto.setModifiedAt(now);
dto.setCreatedAt(now);
dto.setCreatedTs(System.currentTimeMillis());
return boardRepository.save(boardMapper.ItemToEntity(dto));
})
.flatMap(board -> {
dto.setId(board.getId());
Object encode;
try {
encode = objectMapper.writeValueAsBytes(board);
} catch (JsonProcessingException e) {
return Mono.error(new RuntimeException(e));
}
return redisZSet.add("board", encode, dto.getScore())
.flatMap(bool -> {
if (bool) {
return Mono.just(dto);
} else {
return Mono.error(new RuntimeException("redis zset add error"));
}
});
})
;
}
@Override
public Flux<BoardDocumentDTO.Create> SelectCacheList(Integer page, Integer size) {
return redisZSet.reverseRangeByScore(
"board",
Range.unbounded(),
Limit.limit().offset((page - 1) * size).count(size)
)
.flatMap(object -> {
try {
return Mono.just(objectMapper.readValue((byte[]) object, BoardDocumentDTO.Create.class));
} catch (Exception e) {
return Mono.error(new RuntimeException(e));
}
})
.switchIfEmpty(
boardRepository.findAll().flatMap(board -> {
return Mono.just(boardMapper.ToCreateDTO(board));
})
);
}
}3. 벤치마크 기준
1) 벤치마크 부하
locust -H http://127.0.0.1:9090 -u1000 -r 200 -t 60s --autostart -f tests/locust_board_select.py2) 쓰기 데이터
아래와 같은 양식의 데이터를 랜덤으로 생성해서 무작위 삽입한다.
{
"member_id": {
"$numberLong": "60992"
},
"score": {
"$numberLong": "3"
},
"title": "Elizabeth Norman",
"body": "These religious theory thousand company girl long environment. Party family type. Phone according cup exist.\nService require network real somebody view. Sing woman five by.",
"address": "7321 Michael Roads Apt. 058\nSmithfurt, CT 85072",
"country": "Papua New Guinea",
"job": "Product manager",
"credit": "4716296968179136",
"ssn": "333-96-3366",
"company": "Diaz and Sons",
"phone_number": "+1-610-354-4229x154",
"modified_at": {
"$date": "2023-11-28T01:15:04.126Z"
},
"created_at": {
"$date": "2023-11-28T01:15:04.126Z"
},
"created_ts": {
"$numberLong": "1701134104126"
},
"_class": "com.example.webfluxapi.entity.BoardDocument"
}WRITE DATA
3) 읽기 데이터
아래와 같은 양식의 데이터를 다소 무거운 16.9KB 정도의 크기로 만들어 목록으로 반환한다.
{
"code": 200,
"message": "success",
"data": [
{
"id": "65653f5000b66559c04fe767",
"score": 120000,
"member_id": 71642,
"title": "Jamie Terrell",
"body": "Five move if necessary final mind. Seem stop sound successful.\nYard positive clear beat building American whatever education.",
"address": "98886 Johnson Squares Suite 099\nNorth Christophershire, VA 12430",
"country": "Iran",
"job": "Automotive engineer",
"credit": "38926412620637",
"ssn": "177-97-5494",
"company": "Barnes Ltd",
"phone_number": "941-836-2530x59041",
"modified_at": "2023-11-28 10:16:00",
"created_at": "2023-11-28 10:16:00",
"created_ts": 1701134160136
}, (생략)
]
}LIST DATA
4. JSON, 벤치마크 결과
1) JSON, 저장 용량
2) JSON, 쓰기 성능
3) JSON, 읽기 성능
5. MSGPACK, 벤치마크 결과
1) MSGPACK, 저장 용량
2) MSGPACK, 쓰기 성능
3) MSGPACK, 읽기 성능
6. 결과
JSON 으로 저장하는 것보다 MSGPACK 으로 저장하는 것이 더 효율이 좋고 MSGPACK 데이터를 읽어서 처리하는 기준에서도 조금 더 우세한 것으로 보인다.