Competing guild groups are matched based on the number of participants. You can join the race even if you are the only member of your guild, and you will generally be matched against other guilds with a similar number of active racers.
In rare cases, however, groups may be formed with a slight difference in participant numbers. For example, a guild with 4 racers may be matched against a guild with 5 racers. This is an part of the matchmaking system and does not automatically put the smaller guild at a disadvantage.
Race results depend on many factors, including the number of race points earned, teamwork and strategy. A guild with fewer participants can still perform very well and achieve excellent results.
And don't forget — regardless of your final position in the race, you can still earn rewards by completing race goals and unlocking teamwork rewards.