Designing the courtyard is somewhat the same designing small gardens- but this will only apply to indoor courtyards as outdoor courtyards tend to be bigger.
Generally, the first thing that comes to mind for decorating walls and vertical space is to cover them in climbing vines. While this has a lot of good character, it only decorates the existing boundary without creating a depth or illusion of boundary.
In most cases I’ve found that creating an independent source of height in front of courtyard walls and /or in corners can create a 3d effect much like that found in paintings. In a sense, it makes the independent element a focal point while using the decorated walls as a backdrop or framework. Framing an element this way helps create depth of boundary behind that element.