This really depends on what you’re using it for.
- Do you want to store 8 bit, 16 bit (float) or 32 bit (float) per channel?
- Should the compression be lossy or lossless?
- Do you care about encoding or decoding speed?
- Do you need associated/premultiplied alpha or unassociated/straight alpha?
You cannot compare JPG to EXR, because JPG can only store 8 bit per channel. It stores RGB values (3 channels) and it uses lossy compression. In comparison, with the default settings in Blender, EXR will write 32 bit float values for each channel, it stores RGBA (4 channels) and it uses lossless compression.
Also keep in mind that size on disk does not equal size in memory. The file needs to be decompressed to be used for rendering. Assuming you are using two formats that use lossless compression and have the same bit depth, then it should roughly use the same amount of memory for the same content even if they have a different size on disk because one format has a better compression.
Generally EXR is a good format. You can also use 16 bit and lossy compression (DWAA) if you need to keep the file size really small on disk. TIFF and PNG work too. The latter uses straight alpha though and encoding speed is not great.