We study information theoretic limits of recovering an unknown n dimensional, complex signal vector x _* with unit norm from m magnitude-only measurements of the form y _i = |(Ax _* ) _i | ² , i = 1, 2 ..., m, where A is the sensing matrix. This is known as the Phase Retrieval problem and models practical imaging systems where measuring the phase of the observations is difficult. Since in a number of applications, the sensing matrix has orthogonal columns, we model the sensing matrix as a subsampled Haar matrix formed by picking n columns of a uniformly random m X m unitary matrix. We study this problem in the high dimensional asymptotic regime, where m, n → ∞, while m/n → δ with δ being a fixed number, and show that if m <; (2 - o _n (1)) · n, then any estimator is asymptotically orthogonal to the true signal vector x _* . This lower bound is sharp since when m > (2 + o _n (1)) · n, estimators that achieve a non trivial asymptotic correlation with the signal vector are known from previous works.