I've got a Brute Vigilante who also enjoys taking an Enlarge Person potion now and then (we've ruled that the brute ability is not magical and can stack with the magical enlargement).

I've read and implemented the rules on squeezing, but that doesn't deter him (and that's fine), he enjoys being The Hulk.

Now, the problem arises when the spellcaster, a magus, needs to get through to join the combat and cast a spell. I've implemented squeezing constraints on the combat (-4 to AC, -4 to hit), which also means the Magus' turn cannot end in the same square as the Brute (a pain for the magus as this would make Spellstrike more effective/easier). The question then is, can the magus squeeze into a shared space (with a 5-foot step) and use spell combat?

If the typical Medium magus wants to occupy a square currently occupied by the typical Huge vigilante, then the vigilante must somehow vacate that square

On Moving through a Square on Friend says, "You can move through a square occupied by a friendly character…" (emphasis mine), and on Ending Your Movement says, "You can’t end your movement in the same square as another creature unless it is helpless." An exception is made for a Very Small Creature that says, "A Fine, Diminutive, or Tiny creature can move into or through an occupied square" (emphasis mine), but this likely doesn't apply here. Add to these points the rules on Square Occupied by Creature Three Sizes Larger or Smaller that says

Any creature can move [not into and but only] through a square occupied by a creature three size categories larger than itself. A big creature can move through a square occupied by a creature three size categories smaller than it is.

With all this in mind, there's just no way—even if squeezing (see Terrain and Obstacles for more)—for the typical Medium magus (or other typical Medium creature) to occupy a square that's also occupied by the typical Huge vigilante (or other typical Huge, Large, Medium, or Small creature). (A normally Small magus that was the subject of a reduce person spell could also occupy a square occupied by a currently Huge vigilante, though, but that's a long way to go.)

So the typical Medium magus can't move into and remain in a square occupied the typical Huge vigilante; the vigilante must first exit that square somehow. The vigilante can exit the square by squeezing, but no amount of squeezing on the part of the magus's changes the magus's size category, and squeezing alone certainly can't diminish the magus's size by the two categories necessary to render him virtually Tiny!

In short, get a Tiny or littler magus, find an exception that allows what you want (a belt of the weasel may help), make a house rule, or tell the big guy, "Suck in you gut!" The normal rules—sadly—just don't allow what the question describes.

Just a heads-up: Multiple size increases

Your table's ruling that the size increase of the extraordinary ability brute form of the brute archetype vigilante is nonmagical is accurate: that size change effect is an extraordinary ability rather than spell-like or supernatural, and, therefore, an enlarge person spell cast on the brute vigilante in brute form should—according to the Pathfinder treeware—have its normal effect and make the vigilante bigger. The enlarge person spell says, "Multiple magical effects that increase size do not stack" (emphasis mine). (Further, only "other spells that change your size have no effect on you while you are under the effects of a polymorph spell," and, to be clear, neither the brute form effect nor the enlarge person spell is tied to the polymorph subschool.) However, there's also this FAQ exchange:

Size increases and effective size increases: How does damage work if I have various effects that change my actual size, my effective size, and my damage dice?

As per the rules on size changes, size changes do not stack, so if you have multiple size changing effects (for instance an effect that increases your size by one step and another that increases your size by two steps), only the largest applies.…

That these mysterious and sweeping general "rules on size changes" don't actually exist except in this FAQ entry may be (although, in this reader's opinion, shouldn't be) beside the point: While your table can do whatever it wants—so far as I'm aware, Paizo has yet to create a secret police force—, your buddy may not be able to do that (Medium humanoid + brute form + enlarge person = Huge size) thing in another GM's campaign, no matter what the extraordinary ability brute form and the spell enlarge person say.

No

If I understand the question correctly, the Magus still ends up in the Vigilante's space. And as you said yourself:

A creature can squeeze past a creature while moving but it can't end its movement in an occupied square.

It is also implied that squeezing always happens during movement, which means the player cannot stand still and take an action.

Although you could rule that as long as the Magus can move outside of the Vigilante's space after the action, it is able to anything while in its space. Although, that is entirely up to DM Fiat.