Neural networks and in general subsymbolic learning approaches perform well on usual learning tasks, but they are black boxes lacking desired properties such as explainability or trustworthiness. Fully integrated symbolic/subsymbolic systems account for these properties, e.g., through the injection of symbolic information into a subsymbolic learning framework. Systems relying on embeddings can be considered as specific examples of such systems where facts, expressed in some logic, are embedded into a continuous space. Instances mentioned in the facts are mapped to points and the concepts mentioned in the facts are mapped to a region in that space. This allows to exploit geometric regularities in order to tackle typical learning tasks such as link prediction. Embeddings provide the first step towards filling the gap between qualitative, Tarskian style semantics which is used for deductive reasoning over the facts, and quantitative structures which are used for representing objects, relations, and concepts for learning purposes. However, to enable meaningful reasoning, embeddings of relations and concepts are not allowed to be shaped arbitrarily. Especially, convex sets turned out to be appropriate due to their computational advantages and due to their foundation in cognition. Convexity can be defined with a ternary betweenness relation and concepts can be defined as betweenness-closed (= convex) sets. Though many interesting phenomena of cognitive reasoning can be explained in such a framework, it is at least not obvious how to use betweenness for other, more logico-formal aspects of reasoning that, e.g., require defining logical operators. However, convexity is not the only relevant notion for embedding approaches. Next to it, also similarity is a notion vital for inference and allowing for modeling negation in form of dissimilarity. This leads to the possibility of modeling negation via the orthoframes of Goldblatt (1974). This dissertation tackles the question of how these two fundamental aspects of embedding approaches can be combined. In this dissertation, I provide results for connecting betweenness and similarity, thus convexity and orthoframes. In particular, I investigate the construction of a space of convex concepts equipped with an orthogonality relation. I give a universal construction of a betweenness relation over an orthoframe and show that based on Euclidean betweenness (thus using the classical notion of convexity) and under some natural re- strictions, convex cones (and a relaxation thereof) are the only structures capable of modeling such a space. Next to the propositional case, also the extension to modeling relations in an expressive way is considered, on the one hand based on reification allowing for modeling expressive partial relations, on the other hand focused on modeling an arbitrary quantifier depth. For increasing explainability and trustworthiness, it is not only necessary to have an interpretable approach but also to be able to determine its expressivity. Therefore, logical commitments are introduced as a way to determine the expressivity of embedding approaches.