Theta lifting of holomorphic discrete series

The case of U(p, q) × U(n, n)

Kyo Nishiyama

Faculty of Integrated Human Studies

Kyoto University Sakyo, Kyoto 606-8501, Japan

 E-mail: kyo@math.h.kyoto-u.ac.jp

Chen-bo Zhu

Department of Mathematics National University of Singapore Kent Ridge Singapore 119260

                                         E-mail: matzhucb@nus.edu.sg

Compiled on April 20, 2000, Ver. 1.0 [Thu Apr 20 18:05:43 JST 2000]

Contents

1 Moment maps 5

2 Null cone 7

3 Theta lift of orbits 8 3.1 Theta lift of the trivial orbit . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Theta lift of the dense holomorphic orbit . . . . . . . . . . . . . . . . . . . 9

4 Theta lifting associated to the dual pair (U(n, n), U(p, q)) 11 4.1 Howe’s maximal quotient . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.2 Explicit K̃-type formulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4.3 Theta lifting and associated cycles . . . . . . . . . . . . . . . . . . . . . . . 16

Abstract

Let (G, G′) = (U(n, n), U(p, q)) (p + q ≤ n) be a reductive dual pair in the stable range. We investigate theta lifts to G of unitary characters and holomorphic discrete series representations of G′, in relation to the geometry of nilpotent orbits. We give explicit formulas for their K-type decompositions. In particular, for the theta lifts of unitary characters, or holomorphic discrete series with a scalar extreme K′-type, we show that the K structure of the resulting representations of G is almost identical to the K_{ℂ}-module structure of the regular function rings on the closure of the associated nilpotent K_{ℂ}-orbits in 𝔰, where 𝔤 = 𝔨 ⊕ 𝔰 is a Cartan decomposition. As a consequence, their associated cycles are multiplicity free.

Introduction

Let 𝔾 = Sp(2N, ℝ) be a real symplectic group of rank N. A pair of subgroups G and G′ is called a dual pair if G′ is the full centralizer of G in 𝔾 and vice versa. We call the pair (G, G′) a reductive dual pair if both G and G′ are reductive. In this paper, we will be mainly concerned with the reductive dual pair

(G, G′) = (U(n, n), U(p, q)) ⊂ 𝔾 = Sp(2N, ℝ),  (0.1)

where N = 2n(p + q).

Let us consider the non-trivial double cover 𝔾̃ = Mp(2N, ℝ) of 𝔾, called the metaplectic group. For a subgroup L of 𝔾, we denote the pullback of L in 𝔾̃ by L̃. The metaplectic group 𝔾̃ has a distinguished unitary representation Ω, which has various names appearing in various references; it is called the oscillator representation, or sometimes the metaplectic representation, the Weil (or Segal-Shale-Weil) representation, etc. Ω is very small, and in fact, its two irreducible constituents are among the four minimal representations of 𝔾̃ attached to the minimal (non-trivial) nilpotent orbit.

Using the oscillator representation Ω, for a given irreducible admissible representation π′ of G̃′, Howe associates π′ with an irreducible admissible representation π of G̃ called the theta lift of π′ ([4]). We shall denote this as π = θ(π′) in this paper. Roughly saying, π is the theta lift of π′ if and only if there is a non-trivial morphism

Ω ⟶ π ⊗ π′ as a (𝔤, K̃) × (𝔤′ × K̃′)-module,  (0.2)

where 𝔤 (respectively 𝔤′) is the complexification of the Lie algebra of G (respectively G′) and K (respectively K′) is a maximal compact subgroup of G (respectively G′). The morphism in (0.2) should be interpreted in the sense of Harish-Chandra modules. For a precise definition of θ(π′), see §4.1.

Assume that the pair (G, G′) is in the stable range with G′ the smaller member (see [1, §5] for its definition). For (G, G′) = (U(n, n), U(p, q)), this assumption amounts to p + q ≤ n.

Under the stable range assumption, the structure of the theta lift θ(χ) of a unitary character χ of G̃′ has been thoroughly investigated for various pairs (see, for example, [8], [9], [15], [18] and [5]). In particular, θ(χ) is K̃-multiplicity-free, and it often has embeddings into certain degenerate principal series representations. It also has relatively small (and explicitly specified) Gelfand-Kirillov dimension. It is likely that these representations should play an important role in the classification theory of unitary representations of classical groups over ℝ.

In this paper, we first give a brief account of some of these properties of θ(χ) by applying geometric considerations on certain nilpotent orbits (for the pair (G, G′) = (U(n, n), U(p, q)), p + q ≤ n). Our geometric approach has several advantages to the other methods. One of the advantages is that we can determine the associated cycle of θ(χ) almost immediately. Another advantage is that the method works equally well for theta lifts of some irreducible admissible representations other than characters. As a typical example, we examine the theta lift of a holomorphic discrete series representation with a scalar extreme K̃′-type.

Let π′_{hol} be a holomorphic discrete series representation of G̃′ = U(p, q)^{∼}. Although π′_{hol} itself is fairly well understood, it is not so for its theta lift θ(π′_{hol}). By the general arguments of Adams [1, §5], most of θ(π′_{hol}) is realized as a derived functor module A_{𝔮}(λ), and consequently, its associated variety can be explicitly described. Furthermore, the Blattner type formula for multiplicity of K-types of A_{𝔮}(λ) will then give the decomposition of θ(π′_{hol})|K̃. However, it is well-known that these general formulas are not very practical; for example, the Blattner type formula for K-types gives the multiplicity as a summation over certain Weyl group, and it is often difficult to extract precise value from it.

In contrast, our method gives θ(π′_{hol})|K̃ completely in terms of the branching coefficients of finite dimensional representations of general linear groups, called Littlewood-Richardson coefficients, and there are known algorithms to calculate them effectively. Moreover our method implies in a straightforward way that the associated cycle of θ(π′_{hol}) is multiplicity free if π′_{hol} has a scalar extreme K̃′-type.

We shall be more precise in the following.

Let 𝔤 = 𝔤𝔩_{2n}(ℂ) be the complexified Lie algebra of G = U(n, n). Take a maximal compact subgroup K = U(n) × U(n) of G. Then it determines a (complexified) Cartan decomposition 𝔤 = 𝔨 ⊕ 𝔰. We denote by N (𝔰) the cone of nilpotent elements in 𝔰. Then the complexification K_{ℂ} = GL_{n} × GL_{n} of K acts on N (𝔰) with finitely many orbits. We use similar notations for G′ = U(p, q).

Assume that (G, G′) = (U(n, n), U(p, q)) is in the stable range with G′ the small member, i.e., p + q ≤ n, and take a nilpotent K′_{ℂ}-orbit O′ ⊂ N (𝔰′). Then we can define the theta lift O = θ(O′) of O′ in terms of certain geometric quotient maps with respect to the action of K_{ℂ} and K′_{ℂ} (see §1). It turns out that the associated variety of θ(χ) (χ = det^{k} for some k ∈ ℤ ) is the theta lift of the trivial orbit {0} ⊂ N (𝔰′), which we denote by O_{p,q}^{1} = θ({0}). Its Jordan type is 2^{p+q} · 1^{2(n−(p+q))}.

We introduce some notations. Denote by Λ_{n}^{+} the set of dominant integral weights for U(n) or GL_{n}:

Λ_{n}^{+} = {λ = (λ_{1}, …, λ_{n}) ∈ ℤ^{n}, λ_{1} ≥ ⋯ ≥ λ_{n}}.

For λ = (λ_{1}, …, λ_{n}) ∈ Λ_{n}^{+}, τ_{λ} denotes an irreducible finite dimensional representation of GL_{n} with highest weight λ, and λ^{∗} = (−λ_{n}, …, −λ_{1}) denotes the highest weight of the contragradient representation τ_{λ}^{∗}. We denote the set of all partitions of length k by P_{k}. If k ≤ n, then P_{k} may be considered as a subset of Λ_{n}^{+} by adding n − k zeros in the tail. Denote 𝕀_{k} = (1, …, 1) ∈ P_{k}. For α ∈ P_{p} and β ∈ P_{q}, we put

α ⊙ β = (α, 0, …, 0, β^{∗}) ∈ Λ_{n}^{+}.  (0.3)

Note that the regular function ring ℂ[\overline{O_{p,q}^{1}}] of the closure of O_{p,q}^{1} inherits naturally a K_{ℂ}-action.

Theorem A The K_{ℂ}-type decomposition of ℂ[\overline{O_{p,q}^{1}}] and the K̃-type decomposition of θ(det^{k}) are multiplicity-free and are described as follows:

ℂ[\overline{O_{p,q}^{1}}] ≃ \begin{matrix}∑ & ⊕ \\ α∈P_{p},β∈P_{q} \end{matrix} (τ_{α⊙β})^{∗} ⊠ τ_{α⊙β},  and

θ(det^{k})|K̃ ≃ \begin{cases}\begin{matrix}∑ & ⊕ \\ α∈P_{p},β∈P_{q} \end{matrix} (τ_{(α+k𝕀_{p})⊙(β+k𝕀_{q})} ⊗ χ_{p,q})^{∗} ⊠ (τ_{α⊙β} ⊗ χ_{p,q}),  k ≥ 0, \\ \begin{matrix}∑ & ⊕ \\ α∈P_{p},β∈P_{q} \end{matrix} (τ_{α⊙β} ⊗ χ_{p,q})^{∗} ⊠ (τ_{(α−k𝕀_{p})⊙(β−k𝕀_{q})} ⊗ χ_{p,q}),  k < 0, \end{cases}

where χ_{p,q} = det^{\frac{p−q}{2}} is a character of U(n)^{∼}. Furthermore, the associated cycle of θ(det^{k}) is given by

AC θ(det^{k}) = [\overline{O_{p,q}^{1}}] (multiplicity-free).

As a corollary of the above theorem, we obtain

Dim θ(det^{k}) = dim O_{p,q}^{1} = (p + q)(2n − (p + q)) and
Deg θ(det^{k}) = deg \overline{O_{p,q}^{1}},

where Dim π denotes the Gelfand-Kirillov dimension of π and Deg π is the Bernstein degree [17].

Next, let us consider a holomorphic discrete series representation π′_{hol} of G′ = U(p, q). Let 𝔰′ = 𝔰′_{+}⊕𝔰′_{−} be a direct sum decomposition of 𝔰′ by Ad K_{ℂ}-invariant spaces. Then the associated variety of π′_{hol} is 𝔰′_{−} = \overline{O′_{hol}} for an appropriate choice of the complex structure. Here O′_{hol} is the open dense K′_{ℂ}-orbit in 𝔰′_{−}. Put Op,q^{hol} = θ(O′_{hol}), the theta lift of O′_{hol}. Then Op,q^{hol} is a 3-step nilpotent orbit with Jordan type 3^{p} · 2^{q−p} · 1^{2n−p−2q} (for q ≥ p).

For μ, ν, λ ∈ Λ_{n}^{+}, define the Littlewood-Richardson coefficient c\begin{matrix}λ \\ μ,ν \end{matrix} by the branching rule:

τ_{μ} ⊗ τ_{ν} ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ_{n}^{+} \end{matrix} c\begin{matrix}λ \\ μ,ν \end{matrix} τ_{λ}.  (0.4)

For m, l ∈ ℤ^{+}, set

\begin{cases}a(m) = m + \frac{n}{2}, b(l) = l + \frac{n}{2},  n even, \\ a(m) = m + \frac{n−1}{2}, b(l) = l + \frac{n+1}{2},  n odd. \end{cases}  (0.5)

Theorem B Let π′_{hol} be a holomorphic discrete series of U(p, q) with the following scalar extreme K′-type:

χ(m, l) = det^{a(m)} ⊠ det^{−b(l)}, m, l ∈ ℤ^{+}.

Then the K_{ℂ} = GL_{n} × GL_{n}-module structure of ℂ[\overline{Op,q^{hol}}] and the K̃-type decomposition of θ(π′_{hol}) are described as follows:

ℂ[\overline{Op,q^{hol}}] ≃ ^{∑⊕} ; \begin{matrix}α,γ∈P_{p} \\ β,δ∈P_{q} \end{matrix} c\begin{matrix}γ⊙δ \\ α,β^{∗} \end{matrix} (τ_{α⊙β})^{∗} ⊠ τ_{γ⊙δ},  and

θ(π′_{hol})|K̃ ≃ ^{∑⊕} ; \begin{matrix}α,γ∈P_{p} \\ β,δ∈P_{q} \end{matrix} c\begin{matrix}γ⊙δ \\ α,β^{∗} \end{matrix} (τ_{(α+a(m)𝕀_{p})⊙(β+b(l)𝕀_{q})} ⊗ χ_{p,q})^{∗} ⊠ (τ_{γ⊙δ} ⊗ χ_{p,q}).

Furthermore, the associated cycle of θ(π′_{hol}) is given by

AC θ(π′_{hol}) = [\overline{Op,q^{hol}}] (multiplicity-free).

As a consequence, we conclude that

Dim θ(π′_{hol}) = dim Op,q^{hol} = (p + q)(2n − (p + q)) + pq,  and
Deg θ(π′_{hol}) = deg \overline{Op,q^{hol}}.

The above results are also valid for the following reductive dual pairs in the stable range

(G, G′) = \{ \begin{matrix}(O(p, q), Sp(2n, ℝ)) & 2n < min(p, q), \\ (U(p, q), U(r, s))) & r + s ≤ min(p, q), \\ (Sp(p, q), O^{∗}(2n)) & n ≤ min(p, q) \end{matrix}

with appropriate modifications. We shall leave this to the interested reader.

Acknowledgments

We would like to thank Peter Trapa, Takuya Ohta for stimulating discussions on the theta lift of nilpotent orbits. The first author thanks National University of Singapore for its warm hospitality during his stay in July 1999 and March 2000; this collaborative work was started in his first stay in 1999. The second author was supported in part by National University of Singapore Academic Research Grant RP3991620.

1 Moment maps

Unless otherwise stated, we always consider a reductive dual pair

(G, G′) = (U(n, n), U(p, q))

in the stable range, where G′ is the smaller member, i.e., we assume that p + q ≤ n throughout in this paper except for §§4.1 and 4.2.

Let 𝔤 = Lie(G)_{ℂ} be the complexified Lie algebra of G and fix a maximal compact subgroup K = U(n) × U(n) in G. It naturally determines a complexified Cartan decomposition 𝔤 = 𝔨 ⊕ 𝔰, which is realized explicitly as

𝔤 = 𝔨 ⊕ 𝔰 = (\begin{matrix}𝔤𝔩_{n} & 0 \\ 0 & 𝔤𝔩_{n} \end{matrix}) ⊕ (\begin{matrix}0 & M_{n} \\ M_{n} & 0 \end{matrix}) ,

where M_{n} = M_{n}(ℂ) denotes the space of all n × n matrices over ℂ. Therefore we can identify 𝔰 = M_{n} ⊕ M_{n}^{∗} = 𝔰_{+} ⊕ 𝔰_{−}. Here, M_{n}^{∗} denote the dual of M_{n} via the trace form (or Killing form). The complexification K_{ℂ} = GL_{n} × GL_{n} of K acts on 𝔰 by the restriction of the adjoint action, and the above notation is also compatible with the action of K_{ℂ}, i.e., 𝔰_{±} are both stable under K_{ℂ}, and 𝔰_{−} = M_{n}^{∗} is the contragredient representation of 𝔰_{+} = M_{n}.

Similarly, we choose a maximal compact subgroup K′ = U(p) × U(q) ⊂ G′, and a complexified Cartan decomposition 𝔤′ = 𝔨′ ⊕ 𝔰′. We identify 𝔰′ = 𝔰′_{+} ⊕ 𝔰′_{−} = M_{p,q} ⊕ M_{q,p}, where M_{p,q} = M_{p,q}(ℂ) denotes the space of all p × q matrices, and we make the identification M_{p,q}^{∗} = M_{q,p} similarly.

We define two moment maps φ and ψ as follows. Put W = M_{p+q,2n} and take

X = (\begin{matrix}x & z \\ y & w \end{matrix}) ∈ W (x, z ∈ M_{p,n}; y, w ∈ M_{q,n}).  (1.1)

Then they are defined as

φ : W → 𝔰,  φ(X) = (^{t}xz, ^{t}(^{t}yw)) = (a, b) ∈ M_{n} ⊕ M_{n},
ψ : W → 𝔰′,  ψ(X) = (x^{t}y, ^{t}(z^{t}w)) = (c, d) ∈ M_{p,q} ⊕ M_{q,p}.

We define an action of K_{ℂ} × K′_{ℂ} on W in such a way that it makes φ and ψ K_{ℂ} × K′_{ℂ}- equivariant maps. Note that K_{ℂ}-action on 𝔰 is given by the adjoint action, while K′_{ℂ}-action on 𝔰 is trivial. Similar remarks are applicable to the action of K_{ℂ} × K′_{ℂ} on 𝔰′.

Let φ^{∗} and ψ^{∗} be the induced algebra homomorphisms of regular function rings. Thus for example, we have the algebra homomorphism φ^{∗} : ℂ[𝔰] → ℂ[W], and in terms of matrix entry coordinates, it is given by

φ^{∗}(a_{ij})(X) = (^{t}xz)_{ij} = ∑_{k=1}^{p} x_{ki}z_{kj}, φ^{∗}(b_{ij})(X) = ^{t}(^{t}yw)_{ij} = ∑_{l=1}^{q} w_{lj}y_{li},

where a_{ij} ∈ ℂ[𝔰_{+}] is the linear functional on 𝔰_{+} taking a = (a_{ij})_{n×n} ∈ 𝔰_{+} to the (i, j)-th entry, and similarly for b_{ij} ∈ ℂ[𝔰_{−}]. Classical invariant theory then tells us that

Image φ^{∗} = ℂ[W]^{K′_{ℂ}} and Image ψ^{∗} = ℂ[W]^{K_{ℂ}}.

This means that both φ and ψ are geometric quotient maps from W onto their images.

For a subset S in 𝔤, we denote by N (S) the subset of nilpotent elements in S. It is known that K_{ℂ} acts on N (𝔰) with finitely many orbits, and that the K_{ℂ}-orbits in N (𝔰) correspond bijectively to G-orbits in N (𝔤_{ℝ}) (Kostant-Sekiguchi correspondence), where 𝔤_{ℝ} denotes the Lie algebra of G (over ℝ). It is easy to see that the moment maps preserve nilpotent elements. Namely we have

φ(ψ^{−1}(N (𝔰′))) ⊂ N (𝔰) and ψ(φ^{−1}(N (𝔰))) ⊂ N (𝔰′).

The following result should be well-known to the experts, and can be proved by explicit matrix calculations, which we shall omit. Recently we have learned more general and sophisticated versions from Takuya Ohta.

Theorem 1.1 Assume that p+q ≤ n. Take a K′_{ℂ}-orbit O′ ⊂ N (𝔰′). Then Ξ = ψ^{−1}(\overline{O′}) is the closure of a single K_{ℂ}×K′_{ℂ}-orbit in W. As a consequence, the variety Ξ is irreducible, hence φ(ψ^{−1}(\overline{O′})) is the closure of a single K_{ℂ}-orbit O ∈ N (𝔰).

Remark 1.2 Note that, if the nilpotent elements in O′ are k-step nilpotent, then those in O are (k + 1)-step nilpotent.

Definition 1.3 We call the K_{ℂ}-orbit O which is open dense in the image φ(ψ^{−1}(\overline{O′})) the theta lift of O′.

Proposition 1.4 Let O be the theta lift of K′_{ℂ}-orbit O′ ⊂ N (𝔰′). Then the closure \overline{O} is a geometric quotient of Ξ = ψ^{−1}(\overline{O′}) by K′_{ℂ}, i.e., \overline{O} ≃ Ξ//K′_{ℂ}. In particular, we have ℂ[\overline{O}] ≃ ℂ[Ξ]^{K′_{ℂ}}.

Proof. Since φ : W → φ(W) is a geometric quotient map, and Ξ is a K′_{ℂ}-stable closed subvariety of W, the proposition follows from the general arguments on geometric quotients. See [12] and [13] for details. Q.E.D.

2 Null cone

Let W = W_{+} ⊕ W_{−} be a decomposition of W, where

W_{+} = \{(\begin{matrix}x \\ y \end{matrix}) ∈ M_{p+q,n}\} and W_{−} = \{(\begin{matrix}z \\ w \end{matrix}) ∈ M_{p+q,n}\}

in the notation of (1.1). Let

ψ_{+} : W_{+} = M_{p+q,n} ∋ (\begin{matrix}x \\ y \end{matrix}) ↦ x^{t}y ∈ M_{p,q} = 𝔰′_{+}

be the restriction of ψ to the “holomorphic” half of W. We put

𝔑_{p,q} = ψ_{+}^{−1}(0) = ψ^{−1}(0) ∩ W_{+}

and call it the null cone. Note that K′_{ℂ} = GL_{p}×GL_{q} and GL_{n}, which is the left component of K_{ℂ} = GL_{n} × GL_{n}, act simultaneously on W_{+}.

Theorem 2.1 (Kostant) Assume that p + q ≤ n. The null cone 𝔑_{p,q} is irreducible, and the defining ideal I(𝔑_{p,q}) is generated by GL_{n}-invariants of positive degree. Moreover, the regular function ring of 𝔑_{p,q} is naturally isomorphic to H_{p,q}, the space of GL_{n}-harmonic polynomials of W_{+}. We have

ℂ[W_{+}] ≃ ℂ[𝔑_{p,q}] ⊗ ℂ[W_{+}]^{GL_{n}} ≃ H_{p,q} ⊗ ℂ[𝔰′_{+}]

as GL_{n} × K′_{ℂ}-modules.

The above theorem tells us that the action of GL_{n} on W_{+} is completely determined by its action on the null cone 𝔑_{p,q}. Thus we are interested in the GL_{n} ×K′_{ℂ}-module structure of ℂ[𝔑_{p,q}] ≃ H_{p,q}. This is described below by the well-known result of Kashiwara and Vergne [6]. See also [3].

We recall some notations from the Introduction. To make it more transparent, we denote the irreducible finite dimensional representation of GL_{n} with highest weight λ ∈ Λ_{n}^{+} by τ_{λ}^{(n)}. Also for α ∈ P_{p} and β ∈ P_{q}, we put α ⊙_{n} β = (α, 0, …, 0, β^{∗}) ∈ Λ_{n}^{+}.

Theorem 2.2 (Kashiwara-Vergne, Howe) Assume that p+q ≤ n. As a GL_{n} ×K′_{ℂ} = GL_{n} × (GL_{p} × GL_{q})-module, we have

ℂ[𝔑_{p,q}] ≃ H_{p,q} ≃ ^{∑⊕} ; \begin{matrix}λ∈P_{p} \\ μ∈P_{q} \end{matrix} τ_{λ⊙μ}^{(n)}^{∗} ⊠ (τ_{λ}^{(p)}^{∗} ⊠ τ_{μ}^{(q)}).

Remark 2.3 The above isomorphism is a graded isomorphism if we assign the grading on the homogeneous component τ_{λ⊙μ}^{(n)}^{∗} ⊠ (τ_{λ}^{(p)}^{∗} ⊠ τ_{μ}^{(q)}) by |λ| + |μ|, where |λ| = ∑_{i} λ_{i} (resp., |μ| = ∑_{j} μ_{j}) is the size of λ (resp., μ). Similar remarks apply to isomorphisms in Theorem 3.2 and Theorem 3.5.

3 Theta lift of orbits

3.1 Theta lift of the trivial orbit

First, we consider the simplest case, where the orbit O′ is trivial. We write Ξ = Ξ_{p,q}^{1} = ψ^{−1}({0}). Then we clearly have

Ξ = Ξ_{p,q}^{1} = 𝔑_{p,q} × 𝔑_{q,p} ⊂ W_{+} × W_{−},

where the null cone 𝔑_{q,p} is defined similarly as 𝔑_{p,q}. Since 𝔑_{p,q} ≃ 𝔑_{q,p} as varieties, and the action of GL_{n} × K′_{ℂ} on 𝔑_{q,p} is dual to that on 𝔑_{p,q}, we sometimes denote 𝔑_{q,p} by 𝔑_{p,q}^{∗}. In particular, we have ℂ[𝔑_{q,p}] ≃ ℂ[𝔑_{p,q}]^{∗} as a representation of GL_{n} × K′_{ℂ}.

We denote the theta lift of the trivial orbit O′ = \{0\} by O_{p,q}^{1}. The closure \overline{O_{p,q}^{1}} is the geometric quotient of the product of null cones by the action of K′_{ℂ}. It is a two-step nilpotent orbit with Jordan normal form represented by the partition 2^{p+q} · 1^{2n−2(p+q)}.

We include the following simple proposition for its intrinsic interest (see [12] and [13]).

Proposition 3.1 Let O_{p,q}^{1} (p + q ≤ n) be the theta lift of the trivial K′_{ℂ}-orbit in N (𝔰′). Then

(1) every two-step nilpotent K_{ℂ}-orbit in N (𝔰) for G = U(n, n) is of the form O_{p,q}^{1} for some p + q ≤ n. Two orbits O_{p_{1},q_{1}}^{1} and O_{p_{2},q_{2}}^{1} generate the same G_{ℂ}-orbits if and only if p_{1} + q_{1} = p_{2} + q_{2}.

(2) The dimension and the closure of the orbit O_{p,q}^{1} are given by

dim O_{p,q}^{1} = (p + q)(2n − (p + q)), \overline{O_{p,q}^{1}} = ∐_{r≤p,s≤q} O_{r,s}^{1}.

(3) The variety \overline{O_{p,q}^{1}} is normal. If p + q < n holds, then we have ℂ[\overline{O_{p,q}^{1}}] = ℂ[O_{p,q}^{1}].

The regular function ring ℂ[\overline{O_{p,q}^{1}}] has a beautiful K_{ℂ} structure, which is described by the following theorem. It can be thought of as a generalization of the well-known decomposition of ℂ[M_{n,n}] as a GL_{n} × GL_{n}-module. We shall use this to determine the associated cycle of the theta lift of a unitary character of G′ to G.

Denote

Λ_{n}^{+}(p, q) = {α ⊙_{n} β | α ∈ P_{p}, β ∈ P_{q}}.  (3.1)

Theorem 3.2 Assume that p + q ≤ n, and let O_{p,q}^{1} ⊂ N (𝔰) be the theta lift of the trivial nilpotent K′_{ℂ}-orbit in N (𝔰′). Then we have

ℂ[\overline{O_{p,q}^{1}}] ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ_{n}^{+}(p,q) \end{matrix} τ_{λ}^{∗} ⊠ τ_{λ},

as a K_{ℂ} = GL_{n} × GL_{n}-module.

Proof. The proof is similar to that of Theorem 3.5 below. See the remark after the proof of Theorem 3.5. Q.E.D.

Remark 3.3 If p + q = n, one can show that ℂ[O_{p,q}^{1}] ≃ ^{∑⊕}_{λ∈Λ_{n}^{+}} τ_{λ}^{∗} ⊠ τ_{λ}. Therefore, ℂ[O_{p,q}^{1}] is strictly larger than ℂ[\overline{O_{p,q}^{1}}] for p + q = n.

3.2 Theta lift of the dense holomorphic orbit

Note that 𝔰′_{−} ⊂ N (𝔰′). Since 𝔰′_{−} is irreducible and K′_{ℂ}-stable, it has the dense open K′_{ℂ}-orbit O′_{hol}, which consists of the matrices in M_{q,p} = 𝔰′_{−} of the maximal possible rank min(p, q).

Let Op,q^{hol} ⊂ N (𝔰) be the theta lift of O′_{hol}. Since 𝔰′_{−} = \overline{O′_{hol}}, we have \overline{Op,q^{hol}} = φ(ψ^{−1}(𝔰′_{−})) by definition. The elements in Op,q^{hol} are three-step nilpotent, and their Jordan normal forms are represented by the partition 3^{p} · 2^{q−p} · 1^{2n−p−2q} for q ≥ p.

We refer the following proposition again to [12] and [13].

Proposition 3.4 Let Op,q^{hol} be the theta lift of the open dense K′_{ℂ}-orbit in 𝔰′_{−}. Then

(1) the closure \overline{Op,q^{hol}} (p + q ≤ n) is a normal variety. The dimension of the orbit is

dim Op,q^{hol} = (p + q)(2n − (p + q)) + pq.

(2) Op_{1},q_{1}^{hol} and Op_{2},q_{2}^{hol} generate the same complex G_{ℂ}-orbit in N (𝔤) if and only if (p_{2}, q_{2}) = (p_{1}, q_{1}) or (q_{1}, p_{1}).

Theorem 3.5 As a K_{ℂ} = GL_{n} × GL_{n}-module, we have

ℂ[\overline{Op,q^{hol}}] ≃ ^{∑⊕} ; \begin{matrix}α,γ∈P_{p} \\ β,δ∈P_{q} \end{matrix} c\begin{matrix}γ⊙δ \\ α,β^{∗} \end{matrix} (τ_{α⊙β})^{∗} ⊠ τ_{γ⊙δ},

where c\begin{matrix}λ \\ μ,ν \end{matrix} denotes the Littlewood-Richardson coefficient defined in (0.4).

Proof. We put Ξ_{p,q} = ψ^{−1}(𝔰′_{−}) ⊂ W. Then, clearly we have Ξ_{p,q} = 𝔑_{p,q} × W_{−}. Since \overline{Op,q^{hol}} is the geometric quotient of Ξ_{p,q} by K′_{ℂ}, we see

ℂ[\overline{Op,q^{hol}}] ≃ ℂ[Ξ_{p,q}]^{K′_{ℂ}} ≃ (ℂ[𝔑_{p,q}] ⊗ ℂ[W_{−}])^{K′_{ℂ}}.

Since W_{−} = M_{p,n}^{∗} ⊕ M_{q,n}, we get (as a GL_{n} × K′_{ℂ}-module)

ℂ[W_{−}] ≃ ℂ[M_{p,n}^{∗}] ⊗ ℂ[M_{q,n}] ≃ ^{∑⊕} ; \begin{matrix}λ∈P_{p} \\ μ∈P_{q} \end{matrix}(τ_{λ}^{(p)} ⊠ τ_{λ}^{(n)}) ⊗ (τ_{μ}^{(q)}^{∗} ⊠ τ_{μ}^{(n)}^{∗})

≃ ^{∑⊕} ; \begin{matrix}λ∈P_{p} \\ μ∈P_{q} \end{matrix}(τ_{λ}^{(p)} ⊠ τ_{μ}^{(q)}^{∗}) ⊠ \begin{matrix}∑ & ⊕ \\ ν∈Λ_{n}^{+} \end{matrix} c\begin{matrix}ν \\ λ,μ^{∗} \end{matrix} τ_{ν}^{(n)} ≃ \begin{matrix}∑ & ⊕ \\ λ,μ,ν \end{matrix} c\begin{matrix}ν \\ λ,μ^{∗} \end{matrix} τ_{ν}^{(n)} ⊠ (τ_{λ}^{(p)} ⊠ τ_{μ}^{(q)}^{∗}).

In the last summation, note that it is sufficient to consider ν = γ ⊙ δ (γ ∈ P_{p}, δ ∈ P_{q}), because otherwise the Littlewood-Richardson coefficient c\begin{matrix}ν \\ λ,μ^{∗} \end{matrix} vanishes. The module structure of ℂ[𝔑_{p,q}] is given in Theorem 2.2, namely,

ℂ[𝔑_{p,q}] ≃ ^{∑⊕} ; \begin{matrix}α∈P_{p} \\ β∈P_{q} \end{matrix} (τ_{α⊙β}^{(n)})^{∗} ⊠ (τ_{α}^{(p)}^{∗} ⊠ τ_{β}^{(q)}).

Taking K′_{ℂ} = GL_{p} × GL_{q}-invariants in the tensor product of these two spaces, we obtain

ℂ[Ξ_{p,q}]^{K′_{ℂ}} ≃ \begin{matrix}∑ & ⊕ \\ α,β,ν \end{matrix} c\begin{matrix}ν \\ α,β^{∗} \end{matrix} (τ_{α⊙β}^{(n)})^{∗} ⊠ τ_{ν}^{(n)},

where ν = γ ⊙ δ as mentioned above. Q.E.D.

Remark 3.6 We comment on the proof of Theorem 3.2. Recall that \overline{O_{p,q}^{1}} = φ(Ξ_{p,q}^{1}), where Ξ_{p,q}^{1} = 𝔑_{p,q} × 𝔑_{q,p}. Thus we get

ℂ[\overline{O_{p,q}^{1}}] = ℂ[Ξ_{p,q}^{1}]^{K′_{ℂ}} ≃ (ℂ[𝔑_{p,q}] ⊗ ℂ[𝔑_{p,q}]^{∗})^{K′_{ℂ}} .

The rest of the proof remains the same as above.

4 Theta lifting associated to the dual pair (U(n, n), U(p, q))

4.1 Howe’s maximal quotient

Let (G, G′) ⊆ 𝔾 = Sp(2N, ℝ) be a reductive dual pair, and Ω be a fixed oscillator representation of 𝔾̃, the metaplectic cover of 𝔾. Often when no confusion should arise, we shall not distinguish Ω with its Harish-Chandra module.

Denote by Irr(𝔤′, K̃′) the infinitesimal equivalent classes of irreducible admissible (𝔤′, K̃′)-modules, and R(𝔤′, K̃′; Ω) the subset of those in Irr(𝔤′, K̃′) which can be realized as quotients by (𝔤′, K̃′)-invariant subspaces of Ω. According to [4], for each π′ ∈ R(𝔤′, K̃′; Ω) there exists a quasi-simple admissible (𝔤, K̃)-module Ω(π′) of finite length satisfying

Ω/I(π′) ≃ Ω(π′) ⊗ π′,

where

I(π′) = ∩_{ϕ∈Hom′} Ker (ϕ), Hom′ = Hom_{(𝔤′,K̃′)}(Ω, π′).

Furthermore Ω(π′) has a unique irreducible quotient, denoted by θ(π′). Ω(π′) is called Howe’s maximal quotient of π′, and θ(π′) the theta lift of π′.

Note that any ϕ ∈ Hom_{(𝔤′,K̃′)}(Ω, π′) factors through a map ϕ̃ : Ω/I(π′) ↦ π′ and will therefore define an element in the algebraic dual of Ω(π′). This association is (𝔤, K̃)- equivariant, and so we have (by taking K̃-finite vectors)

Lemma 4.1 We have the isomorphism

Ω(π′)^{∗} ≃ Hom_{(𝔤′,K̃′)}(Ω, π′)_{K̃-finite},

where Ω(π′)^{∗} is the dual in the category of Harish-Chandra modules.

We specialize to the case (G, G′) = (U(n, n), U(p, q)) ⊆ Sp(4n(p + q), ℝ). We shall need to use the following

Proposition 4.2 Let (G, G′) = (U(n, n), U(p, q)). Suppose π′ is the (𝔤′, K̃′)-module of

(1) a unitary character and p + q ≤ n; or

(2) a discrete series representation of U(p, q) and p + q ≤ 2n,

then the maximal quotient Ω(π′) is irreducible. Hence we have θ(π′) = Ω(π′).

Proof. (1) is a special case of Proposition 2.1 of [18]. (2) follows from (the proof of) Proposition 2.4 of [11], where it is shown that θ(π′)⊗π′ occurs as an irreducible summand of Ω|_{G̃·G̃′}. Q.E.D.

We consider the see-saw pair ([7, 2]) :

G = U(n, n)  U(p, q) × U(p, q) = L′ ∪ ×  ∪ diagonal K = U(n) × U(n)  U(p, q)  = G′

By the functoriality of the oscillator representation, we have

Ω ≃ ω ⊗ ω^{∗}

as K̃ × L̃′-modules, where ω is an oscillator representation associated to the dual pair:

(U(n), U(p, q)) ⊆ Sp(2(p + q)n, ℝ),

and the first factor U(n) of K acts on the first factor of ω ⊗ ω^{∗} via ω, while the second factor U(n) of K acts on the second factor of ω ⊗ ω^{∗} via the dual of ω.

Recall that associated to the dual pair (U(n), U(p, q)), the covering U(n)^{∼} → U(n) splits if and only if p + q is even. When p + q is odd, U(n)^{∼} can be identified with the half determinant cover, namely

U(n)^{∼} = {(u, c) ∈ U(n) × ℂ^{×}|c^{2} = det(u)}.

Let det^{\frac{1}{2}} be the character of U(n)^{∼} defined by U(n)^{∼} ∋ (u, c) ↦ c. Similar notations apply to the characters of U(p, q)^{∼}.

Let

Λ_{n}^{+}(p, q) = {λ = α ⊙_{n} β ∈ Λ_{n}^{+}|α ∈ P_{k}, β ∈ P_{l}, k ≤ p, l ≤ q, k + l ≤ n}.

Note that this definition coincides with our previous notation of Λ_{n}^{+}(p, q) for p + q ≤ n (see (3.1)).

The decomposition of an oscillator representation for a pair of compact type is wellknown [6, 4]. For the pair (U(n), U(p, q)), we have

ω ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ_{n}^{+}(p,q) \end{matrix}(τ_{λ}^{(n)} ⊗ χ_{p,q}) ⊠ L(τ_{λ}^{(n)}) (as a U(n)^{∼} × U(p, q)^{∼}-module),  (4.1)

where χ_{p,q} = det^{\frac{p−q}{2}} . The irreducible representation L(τ_{λ}^{(n)}) is a unitary highest weight module of U(p, q)^{∼} with minimal K̃′-type (τ_{α}^{(p)}⊗det^{\frac{n}{2}} )⊠(τ_{β}^{(q)}⊗det^{\frac{n}{2}} )^{∗}. Thus as K̃×(𝔤′, K̃′)- modules, we have

\begin{aligned}Ω ≃ ω ⊗ ω^{∗} & ≃ (\begin{matrix}∑ & ⊕ \\ λ∈Λ_{n}^{+}(p,q) \end{matrix}(τ_{λ}^{(n)} ⊗ χ_{p,q}) ⊠ L(τ_{λ}^{(n)})) ⊗ (\begin{matrix}∑ & ⊕ \\ μ∈Λ_{n}^{+}(p,q) \end{matrix}(τ_{μ}^{(n)} ⊗ χ_{p,q})^{∗} ⊠ L(τ_{μ}^{(n)})^{∗}) \\ & ≃ \begin{matrix}∑ & ⊕ \\ λ,μ∈Λ_{n}^{+}(p,q) \end{matrix} ((τ_{λ}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{μ}^{(n)} ⊗ χ_{p,q})^{∗}) ⊠ (L(τ_{λ}^{(n)}) ⊗ L(τ_{μ}^{(n)})^{∗}) . \end{aligned}

In view of the isomorphism Ω(π′)^{∗} ≃ Hom_{(𝔤′,K̃′)}(Ω, π′)_{K̃-finite}, we have the following Proposition 4.3 Let π′ ∈ R(𝔤′, K̃′; Ω). Then

Ω(π′)^{∗}|K̃ ≃

\begin{matrix}∑ & ⊕ \\ λ,μ∈Λ_{n}^{+}(p,q) \end{matrix} dim Hom_{(𝔤′,K̃′)}(L(τ_{λ}^{(n)}) ⊗ L(τ_{μ}^{(n)})^{∗}, π′) ((τ_{λ}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{μ}^{(n)} ⊗ χ_{p,q})^{∗}) ,

or equivalently

Ω(π′)|K̃ ≃

\begin{matrix}∑ & ⊕ \\ λ,μ∈Λ_{n}^{+}(p,q) \end{matrix} dim Hom_{(𝔤′,K̃′)}(L(τ_{λ}^{(n)}) ⊗ L(τ_{μ}^{(n)})^{∗}, π′) ((τ_{λ}^{(n)} ⊗ χ_{p,q})^{∗} ⊠ (τ_{μ}^{(n)} ⊗ χ_{p,q})) .

4.2 Explicit K̃-type formulas

We first discuss some generalities on (𝔤, K)-modules, where 𝔤 is the complexification of the Lie algebra of a semisimple Lie group G and K is a maximal compact subgroup of G.

Let H be a (𝔤, K)-module which is K-admissible, i.e., dim Hom_{K}(H, τ) < ∞ for any τ ∈ Irr(K). We also assume that H is locally K-finite, which means that H = H_{K} where H_{K} denotes the space of K-finite vectors in H. Then, H^{∗} = Hom_{ℂ}(H, ℂ)_{K} is also K-admissible and we have a canonical isomorphism H ≃ (H^{∗})^{∗}. Note that H^{∗} does not denote the algebraic dual of H. A straightforward argument gives

Lemma 4.4 Let H_{1} be a (𝔤, K)-module which is locally K-finite. Then, for any Kadmissible (𝔤, K)-module H_{2} which is locally K-finite, we have

Hom_{(𝔤,K)}(H_{1} ⊗ H_{2}^{∗}, 1) ≃ Hom_{(𝔤,K)}(H_{1}, H_{2}).

By applying Lemma 4.4 twice, we obtain

Corollary 4.5 Let H_{1} be a (𝔤, K)-module which is locally K-finite. Assume that H_{2} and π are K-admissible (𝔤, K)-modules which are locally K-finite, and H_{2}^{∗} ⊗ π^{∗} ≃ (H_{2} ⊗ π)^{∗}. Then we have

Hom_{(𝔤,K)}(H_{1} ⊗ H_{2}^{∗}, π) ≃ Hom_{(𝔤,K)}(H_{1}, H_{2} ⊗ π).

We note that the hypothesis of the above corollary is satisfied if π is a finite dimensional (𝔤, K)-module, or if both H_{2} and π are unitary highest weight modules.

Combining Proposition 4.2, Proposition 4.3 and Lemma 4.4, we obtain the K-type formula of θ(det^{k}) in Theorem A of the Introduction.

Theorem 4.6 (Lee-Zhu) We have

Ω(det^{k})|K̃ ≃ \{ \begin{matrix}^{∑⊕}_{λ∈Λ_{n}^{+}(p,q)} (τ_{λ} ⊗ χ_{p,q})^{∗} ⊠ (τ_{λ} ⊗ χ_{p,q}),  k = 0, \\ ^{∑⊕}_{λ∈Λ_{n}^{+}(p,q)} (τ_{(λ+k𝕀_{p}⊙k𝕀_{q})} ⊗ χ_{p,q})^{∗} ⊠ (τ_{λ} ⊗ χ_{p,q}),  p + q ≤ n, k > 0, \\ ^{∑⊕}_{λ∈Λ_{n}^{+}(p,q)} (τ_{λ} ⊗ χ_{p,q})^{∗} ⊠ (τ_{(λ+|k|𝕀_{p}⊙|k|𝕀_{q})} ⊗ χ_{p,q}),  p + q ≤ n, k < 0. \end{matrix}

In particular, this gives the K̃-type decomposition of the theta-lift θ(det^{k}), for p + q ≤ n.

For η ∈ Λ_{2n}^{+} and μ, ν ∈ Λ_{n}^{+}, define branching coefficients b_{μ,ν}^{η} by

τ_{η}^{(2n)}|_{GL_{n}×GL_{n}} ≃ \begin{matrix}∑ & ⊕ \\ μ,ν∈Λ_{n}^{+} \end{matrix} b_{μ,ν}^{η}τ_{μ}^{(n)} ⊠ τ_{ν}^{(n)}.  (4.2)

The following proposition is a special case of Howe’s reciprocity theorem [2]. We give an argument for the sake of completeness. Similar arguments will be omitted later.

Proposition 4.7 For μ, ν ∈ Λ_{n}^{+}(p, q), we have

L(τ_{μ}^{(n)}) ⊗ L(τ_{ν}^{(n)}) ≃ \begin{matrix}∑ & ⊕ \\ η∈Λ_{2n}^{+}(p,q) \end{matrix} b_{μ,ν}^{η}L(τ_{η}^{(2n)}).

Proof. For the moment, we let (G, G′) = (U(2n), U(p, q)) ⊆ Sp(4n(p + q), ℝ), and let Φ be an associated oscillator representation. We have the see-saw pair:

G =  U(2n)  U(p, q) × U(p, q) = L′ ∪ ×  ∪ diagonal K = U(n) × U(n)  U(p, q)  = G′

Functoriality of the oscillator representation implies that

Φ ≃ ω ⊗ ω

as K̃ × L̃′-modules, where as before ω is an oscillator representation associated to the dual pair:

(U(n), U(p, q)) ⊆ Sp(2(p + q)n, ℝ).

Thus we have

Φ ≃ (\begin{matrix}∑ & ⊕ \\ μ∈Λ_{n}^{+}(p,q) \end{matrix}(τ_{μ}^{(n)} ⊗ χ_{p,q}) ⊠ L(τ_{μ}^{(n)})) ⊗ (\begin{matrix}∑ & ⊕ \\ ν∈Λ_{n}^{+}(p,q) \end{matrix}(τ_{ν}^{(n)} ⊗ χ_{p,q}) ⊠ L(τ_{ν}^{(n)}))

≃ \begin{matrix}∑ & ⊕ \\ μ,ν \end{matrix} ((τ_{μ}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{ν}^{(n)} ⊗ χ_{p,q})) ⊠ (L(τ_{μ}^{(n)}) ⊗ L(τ_{ν}^{(n)})) .  (4.3)

On the other hand, we have (as G̃ × G̃′-modules)

Φ ≃ \begin{matrix}∑ & ⊕ \\ η∈Λ_{2n}^{+}(p,q) \end{matrix}(τ_{η}^{(2n)} ⊗ χ_{p,q}) ⊠ L(τ_{η}^{(2n)}).

From the definition of the branching coefficient (4.2), we have

Φ ≃ \begin{matrix}∑ & ⊕ \\ μ,ν \end{matrix} ((τ_{μ}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{ν}^{(n)} ⊗ χ_{p,q})) ⊠ (\begin{matrix}∑ & ⊕ \\ η∈Λ_{2n}^{+}(p,q) \end{matrix} b_{μ,ν}^{η}L(τ_{η}^{(2n)})) .  (4.4)

Comparing (4.3) and (4.4), we get the desired formula. Q.E.D.

From now on, we assume that the pair (G, G′) is in the stable range with G′ the small member, namely p + q ≤ n. Then L(τ_{λ}^{(n)}) is a holomorphic discrete series for each λ ∈ Λ_{n}^{+}(p, q).

Let χ_{0} be the following character of U(p, q)^{∼}:

χ_{0} = \{ \begin{matrix}1, n even, \\ det^{− \frac{1}{2}} , n odd, \end{matrix}

and for λ ∈ Λ_{n}^{+}(p, q), let

L̃(τ_{λ}^{(n)}) = χ_{0} ⊗ L(τ_{λ}^{(n)}).  (4.5)

Thus L̃(τ_{λ}^{(n)}) defines a true representation of U(p, q).

Recall also that associated to the dual pair (U(n, n), U(p, q)), the covering U(p, q)^{∼} → U(p, q) splits. Note that since we are assuming that the pair is in the stable range, any unitary representation of U(p, q) is in the domain of theta correspondence. See [10].

For λ = α ⊙_{n} β ∈ Λ_{n}^{+}(p, q), denote

λ̃ = α ⊙_{2n} β ∈ Λ_{2n}^{+}(p, q)  (4.6)

by inserting n extra zeroes. We note that each η ∈ Λ_{2n}^{+}(p, q) is of the form λ̃ for some λ ∈ Λ_{n}^{+}(p, q).

Theorem 4.8 Consider the dual pair (G, G′) = (U(n, n), U(p, q)) (p + q ≤ n) in the stable range. Let L̃(τ_{ν}^{(n)}) be a holomorphic discrete series of U(p, q), where ν ∈ Λ_{n}^{+}(p, q). Then its maximal quotient Ω(L̃(τ_{ν}^{(n)})) is irreducible and gives the theta-lift θ(L̃(τ_{ν}^{(n)})) ∈ Irr(U(n, n)^{∼}). We have K̃-type decompositions

θ(L̃(τ_{ν}^{(n)}))|K̃ ≃ \begin{matrix}∑ & ⊕ \\ λ,μ∈Λ_{n}^{+}(p,q) \end{matrix} bλ̃ ; μ,ν(τ_{(λ+ \frac{n}{2} 𝕀_{p}⊙_{n} \frac{n}{2} 𝕀_{q})}^{(n)} ⊗ χ_{p,q})^{∗} ⊠ (τ_{μ}^{(n)} ⊗ χ_{p,q}), n even,

and

θ(L̃(τ_{ν}^{(n)}))|K̃ ≃ \begin{matrix}∑ & ⊕ \\ λ,μ∈Λ_{n}^{+}(p,q) \end{matrix} bλ̃ ; μ,ν(τ_{(λ+ \frac{n−1}{2} 𝕀_{p}⊙_{n} \frac{n+1}{2} 𝕀_{q})}^{(n)} ⊗ χ_{p,q})^{∗} ⊠ (τ_{μ}^{(n)} ⊗ χ_{p,q}), n odd,

where the branching coefficient b_{μ,ν}^{η} is defined in (4.2).

Proof. If n is even, then by Corollary 4.5 and Proposition 4.7, we have

Hom_{(𝔤′,K̃′)}(L(τ_{λ}^{(n)}) ⊗ L(τ_{μ}^{(n)})^{∗}, L(τ_{ν}^{(n)})) ≃ \begin{matrix}∑ & ⊕ \\ η∈Λ_{2n}^{+}(p,q) \end{matrix} b_{μ,ν}^{η} Hom_{(𝔤′,K̃′)}(L(τ_{λ}^{(n)}), L(τ_{η}^{(2n)})).

If η = ξ̃ for ξ ∈ Λ_{n}^{+}(p, q), then we have

L(τ (2n) ; ξ̃) ≃ L(τ_{(ξ+ \frac{n}{2} 𝕀_{p}⊙_{n} \frac{n}{2} 𝕀_{q})}^{(n)}),

by comparing the minimal K̃′-types. Note that when n is odd, we have

L(τ (2n) ; ξ̃) ≃ det^{1/2} ⊗ L(τ_{(ξ+ \frac{n−1}{2} 𝕀_{p}⊙_{n} \frac{n+1}{2} 𝕀_{q})}^{(n)}).

Thus

dim Hom_{(𝔤′,K̃′)}(L(τ_{λ}^{(n)}) ⊗ L(τ_{μ}^{(n)})^{∗}, L(τ_{ν}^{(n)})) = bξ̃ ; μ,ν,

where λ = ξ + \frac{n}{2}𝕀_{p} ⊙_{n} \frac{n}{2}𝕀_{q} and ξ ∈ Λ_{n}^{+}(p, q).

In view of Proposition 4.2 and Proposition 4.3, the desired result follows. The case of odd n is similar. Q.E.D.

4.3 Theta lifting and associated cycles

For the moment, let G be a non-compact semi-simple Lie group of Hermitian type and K a maximal compact group. Let 𝔤 = 𝔨 ⊕ 𝔰 be a complexified Cartan decomposition and further let 𝔰 = 𝔰_{+} ⊕ 𝔰_{−} be the Ad K-stable decomposition of 𝔰. An irreducible unitary representation π of G is said to be holomorphic if there are non-zero K-finite vectors v in the space of π such that π(𝔰_{−})(v) = 0. Then the space of such vectors v is irreducible under K. This is the (unique) minimal K-type of π, and it determines the representation π completely. We denote an irreducible holomorphic unitary representation with the minimal K-type σ ∈ Irr(K) by π(σ).

We give a result on tensor product of a holomorphic unitary representation and a holomorphic discrete series representation with a scalar minimal K-type.

Proposition 4.9 Let π be a holomorphic unitary representation of G, and π(χ) be the holomorphic discrete series representation of G with the one-dimensional minimal K-type χ. Suppose that π has the following K-type decomposition:

π|_{K} ≃ ∑_{τ∈Irr(K)} m(τ)τ,

where m(τ) is the multiplicity of τ. Then

π ⊗ π(χ) ≃ ∑_{τ∈Irr(K)} m(τ)π(τ ⊗ χ).  (4.7)

Proof. Since π(χ) is a holomorphic discrete series with the scalar minimal K-type χ, we see that for any K-type τ of π, π(τ ⊗ χ) is also a holomorphic discrete series representation. Thus we have

π(χ)|_{K} ≃ χ ⊗ S[𝔰_{+}],

and

π(τ ⊗ χ)|_{K′} ≃ (τ ⊗ χ) ⊗ S[𝔰_{+}],

where S[𝔰_{+}] is the symmetric algebra over 𝔰_{+}.

Therefore the left and right hand side of (4.7) are isomorphic as K-modules.

On the other hand, it is clear that π⊗π(χ) is the direct sum of irreducible holomorphic unitary representations (cf. Proposition 4.7 for the case which concerns us), and each Ktype occurs with finite multiplicity. General theory for holomorphic representations tells us that their G-module decompositions (in the Grothendieck group) are determined by the weight space decompositions with respect to the compact Cartan subgroup T ⊆ K. We thus conclude that the isomorphism of the left and right hand side of (4.7) as K-modules in fact induces an isomorphism as G-modules. This proves the proposition. Q.E.D.

We are now back to the dual pair (G, G′) = (U(n, n), U(p, q)) (p + q ≤ n) in the stable range. For m, l ∈ ℤ^{+}, denote

ν(m, l) = m𝕀_{p} ⊙_{n} l𝕀_{q} ∈ Λ_{n}^{+}(p, q).

Then L̃(τ_{ν(m,l)}^{(n)}) is a holomorphic discrete series of U(p, q) with the scalar minimal K′-type

χ(m, l) = \{ \begin{matrix}(det^{\frac{n}{2}} ⊠ det^{− \frac{n}{2}} ) ⊗ (det^{m} ⊠ det^{−l}), n even, \\ (det^{\frac{n−1}{2}} ⊠ det^{− \frac{n+1}{2}} ) ⊗ (det^{m} ⊠ det^{−l}), n odd. \end{matrix}

Proposition 4.10 For μ ∈ Λ_{n}^{+}(p, q), we have

L(τ_{μ}^{(n)}) ⊗ L(τ_{ν(m,l)}^{(n)}) ≃ \begin{matrix}∑ & ⊕ \\ α∈P_{p},β∈P_{q} \end{matrix} c_{α,β^{∗}}^{μ}L(τ_{(α+m𝕀_{p})⊙_{2n}(β+l𝕀_{q})}^{(2n)}).

Proof. We consider the see-saw pair:

∩ ×  ∩_{U(n) × U(n)  U(p, q) = G′}^{U(n)  U(p) × U(q) = K′}

Howe’s reciprocity theorem [2] implies that

L(τ_{μ}^{(n)})|_{K′} ≃ (det^{− \frac{n}{2}} ⊠ det^{\frac{n}{2}} ) ⊗ (\begin{matrix}∑ & ⊕ \\ α∈P_{p},β∈P_{q} \end{matrix} c_{α,β^{∗}}^{μ}τ_{α}^{(p)} ⊠ (τ_{β}^{(q)})^{∗}).

Proposition 4.9 then implies the result. Q.E.D.

By Proposition 4.7 and Proposition 4.10, we have the following

Corollary 4.11 For μ ∈ Λ_{n}^{+}(p, q), and α ∈ P_{p}, β ∈ P_{q}, we have

c_{α,β^{∗}}^{μ} = b_{μ,ν(m,l)}^{(α+m𝕀_{p})⊙_{2n}(β+l𝕀_{q})}, m, l ∈ ℤ^{+}.

Theorem 4.8 and Corollary 4.11 now imply the K̃-type formula of θ(π′_{hol}) in Theorem B of the Introduction.

We recall the notion of associated variety and associated cycle for a Harish-Chandra module V, which are denoted by AV (V) and AC (V) (see for example [14]). By a general result of [16], we have

Lemma 4.12 The associated varieties of the theta lifts θ(det^{k}) and θ(L̃(τ_{ν(m,l)}^{(n)})) are given as

AV (θ(det^{k})) = \overline{O_{p,q}^{1}}, and AV (θ(L̃(τ_{ν(m,l)}^{(n)}))) = \overline{Op,q^{hol}}.

Remark 4.13 The associated variety of the theta lift of any holomorphic discrete series of U(p, q) is the same, namely \overline{Op,q^{hol}}. The same remark is valid for the theta lift of any irreducible finite dimensional unitary representation of U(p, q). Of course such an irreducible finite dimensional unitary representation is a unitary character of U(p, q) unless pq = 0.

The main result of this section is the statement on associated cycles of θ(det^{k}) and θ(L̃(τ_{ν(m,l)}^{(n)})) in Theorems A and B of the Introduction.

Theorem 4.14 Consider the dual pair (G, G′) = (U(n, n), U(p, q)) (p + q ≤ n) in the stable range. Then we have

AC (θ(det^{k})) = [\overline{O_{p,q}^{1}}], and AC (θ(L̃(τ_{ν(m,l)}^{(n)}))) = [\overline{Op,q^{hol}}], k ∈ ℤ, m, l ∈ ℤ^{+}.

Proof. We compare the K̃-module structure of θ(det^{k}) (resp., θ(L̃(τ_{ν(m,l)}^{(n)}))) with the K_{ℂ}-module structure of ℂ[\overline{O_{p,q}^{1}}] (resp., ℂ[\overline{Op,q^{hol}}]).

For det^{k}, the K̃-module structure of θ(det^{k}) coincides with the K_{ℂ}-module structure of ℂ[\overline{O_{p,q}^{1}}] up to the obvious determinant shift, and the shift in the parameter λ = α ⊙_{n} β → (α + k𝕀_{p}) ⊙_{n} (β + k𝕀_{q}). See Theorem 4.6 and Theorem 3.2.

For L̃(τ_{ν(m,l)}^{(n)}), since c_{α,β^{∗}}^{μ} = b_{μ,ν(m,l)}^{(α+m𝕀_{p})⊙_{2n}(β+l𝕀_{q})} for μ ∈ Λ_{n}^{+}(p, q), and α ∈ P_{p}, β ∈ P_{q}, we

see that the K̃-module structure of θ(L̃(τ_{ν(m,l)}^{(n)})) coincides with the K_{ℂ}-module structure of ℂ[\overline{Op,q^{hol}}] up to the obvious determinant shift and the shift in the parameter α ⊙_{n} β → (α + a(m)𝕀_{p}) ⊙_{n} (β + b(l)𝕀_{q}). Here a(m) and b(l) are given in (0.5). See Theorem 4.8 and Theorem 3.5.

Thus in either case, the two Hilbert polynomials (associated to K_{ℂ}-stable filtrations) have the same degrees and the same leading terms. In particular we have

Deg (θ(det^{k})) = deg(\overline{O_{p,q}^{1}}), and Deg (θ(L̃(τ_{ν(m,l)}^{(n)}))) = deg(\overline{Op,q^{hol}}).

Our assertion follows from the equality of these degrees. See [14, Th. 1.4]. Q.E.D.

References

[1] J. Adams, L-functoriality for dual pairs. Orbites unipotentes et représentations, II. Astérisque No. 171-172, (1989), 85–129.

[2] R. Howe, Reciprocity laws in the theory of dual pairs, in Representation Theory of Reductive Groups (ed. P.C. Trombi), Progress in Mathematics 40 (1983), pp159-175, Birkhäuser.

[3] R. Howe, Remarks on classical invariant theory. Trans. Amer. Math. Soc. 313 (1989), no. 2, 539–570. Erratum to: ”Remarks on classical invariant theory”. Trans. Amer. Math. Soc. 318 (1990), no. 2, 823.

[4] R. Howe, Transcending classical invariant theory. J. Amer. Math. Soc. 2 (1989), no. 3, 535–552.

[5] J.-S. Huang and J.-S. Li, Unipotent representations attached to spherical nilpotent orbits, Ameri. Jour. Math. 121 (1999), no. 3, 497-517.

[6] M. Kashiwara and M. Vergne, On the Segal-Shale-Weil representations and harmonic polynomials, Invent. Math. 44 (1978), no. 1, 1–47.

[7] S. Kudla, Seesaw dual reductive pairs, “Automorphic forms of several variables”, Progress in mathematics 46 (1983), 244-268.

[8] S. T. Lee and C.-B. Zhu, Degenerate principal series and local theta correspondence, Trans. Amer. Math. Soc. 350 (1998), no. 12, 5017–5046.

[9] S. T. Lee and C.-B. Zhu, Degenerate principal series and local theta correspondence II, Israel J. Math., 100 (1997), 29 – 59.

[10] J.-S. Li, Singular unitary representations of classical groups, Invent. Math., 97 (1989), 237 – 255.

[11] J.-S. Li, Theta lifting for unitary representations with nonzero cohomology, Duke Math. J., 61 (1990), 913 – 937.

[12] Kyo Nishiyama, Multiplicity-free actions and the geometry of nilpotent orbits. Preprint, 1999.

[13] Kyo Nishiyama, Theta lifting of two-step nilpotent orbits for the pair O(p, q) × Sp(2n, ℝ). To appear in Proceedings of the Symposium on “Infinite Dimensional Harmonic Analysis” (Kyoto, September 1999).

[14] Kyo Nishiyama, Hiroyuki Ochiai, and Kenji Taniguchi, Bernstein degree and associated cycles of Harish-Chandra modules (Hermitian symmetric case), Kyushu University Preprint Series in Mathematics, 1999-16.

[15] E.-C. Tan and C.-B. Zhu, On certain distinguished unitary representations supported on null cones, Amer. J. Math. 120 (1998), no. 5, 1059–1076.

[16] P. Trapa, Annihilators, associated varieties, and the theta correspondence, preprint, November 1999.

[17] D. Vogan, Gelfand-Kirillov dimensions for Harish-Chandra modules, Inventiones Math. 48 (1978), 75-98.

[18] C.-B. Zhu and J.-S. Huang, On certain small representations of indefinite orthogonal groups, Represent. Theory 1 (1997), 190–206 (electronic).