Pandas QST: How to convert large queryset into dataframes in optimized way?

Research

• [X] I have searched the [pandas] tag on StackOverflow for similar questions.

• [X] I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://stackoverflow.com/questions/77222590/how-to-convert-large-queryset-into-dataframes-in-optimized-way

Question about pandas

A django queryset of more than 5 lakhs records when converting into dataframes is getting into OutOfMemory issue. query_data = MyObjects.objects.filter().values() df = pd.DataFrame.from_records(query_data) query_data is returning around 6 lakh rows but pandas is unable to process this much data and got stuck. How can we optimize or prevent this.

Comment From: mroeschke

Thanks but this issue needed more context to be helpful for closing