What is an In-Memory Format?

Last updated Oct 14, 2022 - Edit Source

• Data engineering

In-memory formats are optimized to:

• hit fast instruction sets
• be cache friendly
• be parallelizable

Formats:

• Apache Arrow
• Apache Spark DataFrames
• NumPy
• Pandas

The opposed to in-memory formats are Data Lake File Formats which save space, be cross-language and serve as long-term storage. More on the Data AI Summit talk about From bits to Data Frames by Jorge C Leitao.