I hope you find this read interesting and you are in a good mood like me as today was Holi, the great Indian festival of colors. However, at work the same fun was not there as there was a rush job for fixing an error in some codebase, pufff. It was a python project that was calling rets server to fetch information from different properties and looping within. MongoDB temporary tables were created under this as the data was first getting stored in them before saving it in the actual MongoDB data collection, which was causing the entire server disk space to be used. What optimization I did was to use Redis lists in place of this arrangement which reduced the space requirements and removal of data from redis bears lesser side effects than doing that in temporary collection which gets saved to hard disk on every write operation.
However, this means that If I am using polyglot persistence in my project (as this term is known), then the amount of code complexity to deal with the different data stores increases.
My work on the project remains unfinished as I have to nail getting the proper RETS server data (using their customized querying mechanism) and also find out why and how the existing data was wrongly extracted.
No comments:
Post a Comment