The films emulated in Film 04 can be quite bold and high contrast, so the issue of contrast changing overall brightness that has applied to all of the film packs can especially show itself in Film 04.
Contrast, by nature, is making the brights brighter and the darks darker. If you have a photo of a brightly lit model against a white background, your entire photo is going to be composed of bright values, and hence a contrast adjustment is going to make it all brighter. Conversely, if you have a photo taken at night or in shadows, your entire photo is mostly going to be composed of dark values and a contrast adjustment is going to make it all darker. Since Film 04's contrast is especially high, this effect is more noticeable at times.
The fix is to adjust your exposure slider to compensate for what you're seeing. Here's something to try: before you apply a VSCO Film preset, turn the contrast slider to +100. It will look pretty bad, don't worry. Now, adjust your exposure so that it looks correct, and then apply a VSCO Film 04 preset. You should see a very minimal brightness change doing that.
If you find you have a constant problem with brightness, it may be that you are of the ETTL or ETTR (Expose to the Left/Right) crowd. While there are merits to each philosophy, by shooting that way you will be feeding an inherently dark or bright photo into VSCO Film. To compensate, you'll either need to just apply an exposure adjustment across all of your photos, or adjust your exposure method when you're actually shooting to avoid this hassle.
Be sure to watch this great walkthrough video on VSCO Film 04 here: http://vsco.co/blog/vsco-film-04-walkthrough-video.