We construct a family of right coideal subalgebras of quantum groups, which have the property that all irreducible representations are one-dimensional, and which are maximal with this property. The obvious examples for this are the standard Borel subalgebras expected from Lie theory, but in a quantum group there are many more. Constructing and classifying them is interesting for structural reasons, and because they lead to unfamiliar induced (Verma-)modules for the quantum group. The explicit family we construct in this article consists of quantum Weyl algebras combined with parts of a standard Borel subalgebra, and they have a triangular decomposition. Our main result is proving their Borel subalgebra property. Conversely we prove under some restrictions a classification result, which characterizes our family. Moreover we list for Uq(4) all possible triangular Borel subalgebras, using our underlying results and additional by-hand arguments. This gives a good working example and puts our results into context.