That only works if the Signal user gives others access to their phone number (and you have that number saved with a suitable name in your phone’s contacts).

But kind of defeats the purpose of Grpahene OS.

No, it doesn’t. The sandboxing ensures that the Google services have none of the broad access to the operating system they have on a normal Android. For example Google won’t get access to your location unless you explicitly give the permission.